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Placing  Context  On  The  Fiscal 
Environment 
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•  Future  demographics  demand  more  from  DOD’s  investment  accounts 

•  Current  practices  yield  systems  at  unexpectedly  high  cost,  diminished 
results 
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■  FY  1998  Plan  ■  FY  2005  Plan 


FY  ’05:  $89.95  billion  total 


FY  1998  plan  for  completing^^^ 
development  of  8  programs 


$47.9  B 


Additional  investment 
needed  under  FY  2005 
plan  for  completing  the 
8  programs 


^Source:  Selected  Acquisition  Report  data  (12/31/96  and  12/31/03)  on  the  8  weapon  systems  among  the  highest  R&D  budget  requests  for  FY  2003. 
Note:  All  dollars  are  in  constant  FY  2005  dollars. 
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Delivering  the  Product 
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Program  Launch 


requirements  expected  quality  targets 


Knowledge  Point  -  1 


Knowledge  Point  -  2  Knowledge  Point  -  3 
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The  Knowledge-Based  Approach 
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Technology  development  is  done  separately  and  in  advance  of 
product  development;  Milestone  B  is  the  dividing  iine. 


Knowledge  Point  1:  At  milestone  B,  a  match  is  achieved  between  the 
user’s  needs  and  the  developer’s  resources  (key  indicator: 
technoiogy  readiness  ievei). 


Knowledge  Point  2:  At  Critical  Design  Review,  the  product  design 
demonstrates  its  ability  to  meet  user  needs  and  is  stable  (key 
indicator:  %  of  engineering  drawings  reieased). 


Knowledge  Point  3:  At  milestone  C,  it  is  demonstrated  that  the 
product  can  be  produced  within  cost,  schedule,  and  quality  targets 

(key  indicator:  %  of  key  processes  in  statisticai  controi). 
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DOD  Process  and  Best  Practices 
Model  Has  Similar  Framework 
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Production 

Commitment 
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Knowledge  Point  1 

Technologies  and 
resources  match 
requirements 


Program 

Launch 


Knowledge  Point  2 

Design  performs  as 
expected 


elopment 

Demonstration 

Production 


Knowledge  Point  3 

Production  can 
meet  cost, 
schedule,  and 
quality  targets 
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Knowledge  at  Key  Junctures  is  Critical 
To  Delivering  Programs  As  Promised 
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Knowledge  Points: 


1:  Technology  is  mature 
enough  to  meet 
requirements  (the  match) 

2:  Design  is  mature 

3:  Production  processes  are 
mature 


Unknowns 


Knowns 
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Knowledge  Based  Approach 
Aligns  with  Systems  Engineering 
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Actual  Acquisition  Strategies  Do  Not 
Align  with  Systems  Engineering 
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Milestone  A 
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Knowledge  Point  1 :  Match  Developer 
Resources  With  User  Needs 
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Product  Development 


Technology 

-A 

lAgration 

Demonstration 

Production 

Development 
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_ L_ _ 

A 

Milestone  B  Business  Case  Is  Key 


•  If  a  program  is  unexecutable  within  resources  at  Milestone  B,  negative 
consequences  are  unavoidable 

•  Essential  elements  of  a  sound  business  case: 

•  A  requirement  exists  that  warrants  a  materiel  solution  consistent  with 
national  military  strategy  priorities 

•  The  materiel  developer  has  the  requisite  mature  technologies  and 
technical  knowledge  necessary  to  meet  the  requirement 

•  Systems  engineering  employed  and  preliminary  design  has  been 
established 

•  The  materiel  developer  has  a  knowledge-based  product  development  plan 
that  will  attain  high  levels  of  design  and  production  maturity  at  the  right 
times. 

•  Reasonable  estimates  and  funding  is  available  to  fully  resource  the 
product  development  and  production  plan 
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Immature  Technologies  at  Product  Launchu 
Have  Weakened  DOD  Business  Cases  M 
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•  Achieving  high  ievei  of  maturity  criticai  indicator  that 
business  case  is  executabie. 


•  Beginning  product  deveiopment  with  immature  technoiogies 
iongstanding  DOD  probiem. 


•  Predictabiiity  is  key,  invention  cannot  be  scheduied  and  costs 
difficuit  to  predict. 


•  Cost  of  proceeding  into  system  deveiopment  without 
compieting  technoiogy  deveiopment  can  be  dramatic. 
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Best  Practices  for  Including 
Technology  Onto  Products 
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•  Technology  development  separate  from  product 
development 

•  Responsibility  for  technology  development  to  S&T 
organizations  rather  than  product  managers 

•  Development  follows  disciplined  process 

•  Environment  is  critical  for  technical  maturation 

•  Decision  makers  need  tools  and  authority 


Technology  not  brought  onto  product  unless 
demonstrated  that  it  meets  requirements. 
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Technology  Readiness  Levels  For 
Intended  Product 
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Application 


Applicable 


Feasible 


Possible 


> 


> 


9  Actual  system  “flight  proven”  through 
sueeessful  mission  operations 
8  Aetual  system  eompleted  and  “flight 
qualified”  through  test  and  demonstration 
7  System  prototype  demonstration  in  an 
operational  environment 
6  System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 
5  Component  and/or  breadboard 
validation  in  relevant  environment 
4  Component  and/or  breadboard 
validation  in  laboratory  environment 
3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof-of- 
concept 

2  Technology  concept  and/or  application 
formulated 

1  Basie  prineiples  observed  and  reported 


Using  TRLs  to  Match  Technology  with 
Product  Launch  Requirements 


Product 

Requirements 


TRL 
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High  risk  for  Low  risk  for 

product  launch  product  launch 
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DOD  Has  Made  Constructive  Changes  In  Policy, 
But  Implementation  Remains  A  Challenge 
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•  DOD  5000  policy  says  most  of  the  right  things  about 

separating  technoiogy  deveiopment  from  system  deveiopment 

•  Calls  for  technology  maturity  to  TRL  6  (relevant  environment) 

•  Calls  for  evolutionary  approach  as  a  check  on  reqts. 

•  Short  development  cycle  times  (5  years  or  less) 


•  However, 

•  Best  practice  standard  is  TRL  7  (operational  environment) 

•  Most  individual  programs  do  not  even  abide  by  policy 

•  Many  programs  fall  outside:  satellites,  MDA,  ships 

•  Those  within  are  unique:  eg.,  FCS,  JSF 

•  Preference  is  still  for  revolutionary,  not  evolutionary 

•  Knowledge  gaps  and  optimistic  estimates  at  MS  B  are  the  norm 
and  are  reinforced  with  approval  and  funding 
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Percent  of  Programs  that  Achieved  i  r*  A  n 

Technology  Maturity  at  Key  Junctures  ^ 
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43 


25% 
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Source:  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs. 
GAO-06-391.  Washington,  DC.:  March  2006. 
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Immature  Technologies  Continue  To 
Spill  Over  Into  System  Development 
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Average  Program  Research,  Development,  Test  and 
Evaluation  Cost  Growth  From  First  Full  Estimate 

Percent 
40 

30 

20 

10 

0 

Mature  Technologies  Immature  Technologies 


Source:  Defense  Acquisitions:  Assessments  of  Seiected  Major  Weapon  Programs. 
GAO-06-391.  Washington,  DC.:  March  2006. 
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Condition  Continues  Under  Revised  Policy 
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MMA 
AESA 
JTRS  Cluster  1 
Global  Hawk 
Future  Combat  System 
F-35 
WIN-T 
E-2C  Hawkeye 
JTRS  Cluster  5 
Aerial  Common  Sensor 
EA-18G 

Expeditionary  Fighting  Vehicle 
MEADS 
C-130  AMP 
C-5  RERP 
MP-RTIP 
SDB 
B-2  RMP 


y  Less  than  TRL  6 


TRL  6  or  greater 


50  100  150  200  250 


Percent  Cost  Growth 


Source:  Defense  Acquisitions:  Major  Weapon  Systems  Continue  to  Experience  Cost 

And  Schedule  Problems  Under  DOD’s  Revised  Policy.  GAO-06-368.  Apr.  13,  2006  1  7 


Challenges  That  Remain  To 
Getting  Better  Outcomes 
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•  How  will  initiatives  like  strengthened  focus  on  SE,  EMRLS,  etc.,  succeed 
when  incentives  encourage  starting  programs  too  early,  making  revolutionary 
technical  leaps,  underestimating  cost  and  risk,  and  promising  record  delivery 
times? 

•  Can  we  employ  evolutionary  acquisition  and  trade  requirements  to  match  a 
5-year  cycle  and  to  mature  technologies? 

•  Can  we  provide  S&T  the  funds,  organization,  and  authority  to  do  the 
necessary  pre-acquisition  SE  and  technology  development  work? 

•  Can  we  put  managers  in  a  position  to  succeed  with  a  shorter  cycle  and  hold 
them  accountable  for  results? 

•  Can  capabilities-based  requirements  be  controlled  so  that  tradeoffs  can  be 
made? 

•  If  the  solution  requires  the  invention  of  numerous  technologies,  is  so 
complex  that  the  government  cannot  be  the  integrator,  and  so  expensive  that 
it  takes  most  of  a  service’s  budget,  is  it  really  a  viable  solution? 
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Encourage  Best  Practices  By 
Changing  Incentives 
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'^Use  knowledge-based  approach  for  decision  making 
'^Measure  knowledge  against  high  and  clear  standards 
Separate  technology  development  from  product  development 
'^Realign  technology  responsibilities  and  funding 
'^Discipline  requirements-setting  process  by  demanding  match 
'^Send  signals  through  investments  on  individual  programs 
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GAO  Reports  On-line 
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Reports  Available  at  www.aao.aov 

•  Best  Practices:  Better  Support  of  Weapon  System  Program  Managers  Needed  To 
Improve  Outcomes.  GAO-06-110.  Washington,  D.C.:  November  30,  2005 

•  Defense  Acquisitions:  Stronger  Management  Practices  Needed  to  Improve  DOD’s  Software- 
Intensive  Weapons  Acquisitions.  GAO-04-393.  March  1,  2004 

•  Best  Practices:  Setting  Requirements  Differently  Could  Reduce  Weapon  Systems’  Total 
Ownership  Costs.  GAO-03-57.  February  11,  2003. 

•  Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early  Improves  Acquisition 
Outcomes.  GAO-02-701.  July  15,  2002. 

•  Best  Practices:  Better  Matching  of  Needs  and  Resources  Will  Lead  to  Better  Weapon  System 
Outcomes.  GAO-01-288.  March  8,  2001. 

•  Best  Practices:  A  More  Constructive  Test  Approach  Is  Key  to  Better  Weapon  System 
Outcomes.  GAO-  NSIAD-00-199.  July  31,  2000. 

•  Best  Practices:  DOD  Training  Can  Do  More  to  Help  Weapon  System  Program  Implement  Best 
Practices.  GAO/NSIAD-99-206.  Augustl6, 1999. 

•  Best  Practices:  Better  Management  of  Technology  Development  Can  Improve  Weapon  System 
Outcomes.  GAO/NSIAD-99-162.  July  30, 1999. 

•  Best  Practices:  DOD  Can  Help  Suppliers  Contribute  More  to  Weapon  System  Programs. 
GAO/NSIAD-98-87.  March  17, 1998. 

•  Best  Practices:  Successful  Application  to  Weapon  Acquisition  Requires  Changes  in  DOD’s 
Environment.  GAO/NSIAD-98-56.  February  24, 1998. 

•  Best  Practices:  Commercial  Quality  Assurance  Practices  Offer  Improvements  for  DOD. 
GAO/NSIAD-96-162.  August  26, 1996. 
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